Kernel Fusion for Video Retrieval Tasks

In the context of the annual TRECVID challenge, this paper presents a comprehensive statistical framework for classification of video shots. We first design and analyze a broad set of language and video features. While most teams in this challenge have not been able to improve their baseline language performance by incorporating visual information, we show that a strong vision system can be a major asset. By leveraging techniques that have been developed for visual object recognition we can detect categories for which language alone does not contain any information. We then present an elegant way to integrate information from different sources into a single non-parametric classifier. We compute kernel matrices for different information sources and for a variety of features and use semi-definite programming to learn the optimal kernel matrix. This optimal kernel is a positive linear combination of the prespecified kernel matrices and is determined simultaneously with the decision boundary of the 1-norm soft-margin SVM. Our preliminary results on a reduced data set are encouraging, and suggest that our data fusion approach could outperform most of the usual classification techniques when multiple sources of information are available.